Acoustic-to-Articulatory Mapping With Joint Optimization of Deep Speech Enhancement and Articulatory Inversion Models

We investigate the problem of speaker independent acoustic-to-articulatory inversion (AAI) in noisy conditions within deep neural network (DNN) framework. In contrast with recent results literature, we argue that a DNN vector-to-vector regression front-end for speech enhancement (DNN-SE) can play key role AAI when used to enhance spectral features prior back-end processing. experimented single- and multi-task training strategies DNN-SE block finding latter be beneficial AAI. Furthermore, show coupling producing enhanced an trained on clean outperforms multi-condition (AAI-MC) tested speech. observe 15% relative improvement Pearson's correlation coefficient (PCC) between our system AAI-MC at 0dB signal-to-noise ratio Haskins corpus. Our approach also compares favourably against using conventional DSP (MMSE IMCRA) front-end. Finally, demonstrate utility articulatory downstream application. report significant WER improvements automatic recognition task mismatched based Wall Street Journal corpus (WSJ) leveraging information estimated by over spectral-alone features.